'’j 


N9 3- 26 082 


THE ADULT LITERACY EVALUATOR: AN INTELLIGENT 
COMPUTER-AIDED TRAINING SYSTEM FOR 
DIAGNOSING ADULT ILLITERATES 


Final Report 

NASA/JSC Summer Faculty Fellowship Program-1992 
Johnson Space Center 


Prepared by: 

David B. Yaden, Jr., Ph.D. 

Academic Rank: 

Associate Professor 

University & Department: 

University of Houston 
Department of Curriculum & 
Instruction 

Houston, Texas 77204-5872 

NASA/JSC 


Directorate: 

Information Systems 

Division: 

Information Technology 

Branch: 

Software Technology 

JSC Colleague: 

James A. Villarreal 

Date Submitted: 

August 3, 1992 

Contract No: 

NAG 9-562 


23-1 



ABSTRACT 


An important part of NASA’s mission involves the secondary application of its 
technologies in the public and private sectors. One current application being 
developed is the The Adult Literacy Evaluator, a simulation-based diagnostic tool 
designed to assess the operant literacy abilities of adults having difficulties in 
learning to read and write. Using ICAT system technology in addition to speech 
recognition, closed-captioned television (CCTV), live video and other state-of-the 
art graphics and storage capabilities, this project attempts to overcome the 
negative effects of adult literacy assessment by allowing the client to interact with 
a intelligent computer system which simulates real-life literacy activities and 
materials and which measures literacy performance in the actual context of its 
use. The specific objectives of the project are as follows: (a) To develop a 
simulation-based diagnostic tool to assess adults’ prior knowledge about reading 
and writing processes in actual contexts of application, (b) to provide a profile of 
readers’ strengths and weaknesses, and (c) to suggest instructional strategies and 
materials which can be used as a beginning point for remediation. In the first and 
developmental phase of the project, descriptions of literacy events and 
environments are being written and functional literacy documents analyzed for 
their components. Examples of literacy events and situations being considered 
included interactions with environmental print (e.g., billboards, street signs, 
commercial marquees, storefront logos, etc.), functional literacy materials (e.g., 
newspapers, magazines, telephone books, bills, receipts, etc.) and employment- 
related communication (i.e., job descriptions, application forms, technical 
manuals, memorandums, newsletters, etc.). Each of these situations and materials 
is being analyzed for its literacy requirements in terms of written display (i.e., 
knowledge of printed forms and conventions), meaning demands (i.e., 
comprehension and word knowledge) and social situation. From these 
descriptions, scripts are being generated which define the interaction between the 
student, an on-screen guide and the simulated literacy environment The 
proposed outcome of the Evaluator is a diagnostic profile which will present 
broad classifications of literacy behaviors across die major areas of metacognitive 
abilities, word recognition, vocabulary knowledge, comprehension and writing. 
From these classifications, suggestions for materials and strategies for instruction 
with which to begin corrective action will be made. The focus of the Literacy 
Evaluator will be essentially to provide an expert diagnosis and an interpretation 
of that assessment which then can be used by a human tutor to further design and 
individualize a remedial program as needed through the use of an authoring 
system. 
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Adult Literacy Evaluator: An Intelligent Computer-Aided Training 

System for Diagnosing Adult Illiterates 


I. Introduction 

This report describes the current status, milestones achieved and projected 
schedule of the Adult Literacy Evaluator (ALE). This spin-off project is 
attempting to integrate expert system, speech analysis and virtual reality 
technology developed at NASA into the field of adult literacy with the primary 
goal of creating a tool for literacy assessment with high ecological validity. The 
outcomes of this assessment tool will also include written diagnostic profiles and 
exemplar lessons in remediation strategies to be used by volunteers and staff at 
literacy provider organizations. 

The following sections in this report include descriptions of the current 
project objectives, past and current developments in theory and the expert system 
knowledge base, the conceptual model, an updated project plan and schedule, past 
presentations and publications by the project team, external funding efforts and 
contractual research in speech analysis and recognition to be incorporated into the 
ALE. 


Finally, the scope of this report covers the time period of March 1991 
through June 1992 and describes the project's development under the current 
principal investigator. Dr. David Yaden. Descriptions of the work effort prior to 
March 1991 are not included in this document. 

II. Current Project Objectives 

The project objectives as of 1991 have remained constant and are as 
follows: 

1. To develop a simulation-based diagnostic tool to assess adults' prior 
knowledge about reading and writing processes in the context of their use; 

2. To provide interpretive, diagnostic profiles of readers' strengths and 
weaknesses; and 

3. To provide focused suggestions (i.e., exemplar lessons) for 
instructional strategies and materials which can be used by volunteer 
tutors as a beginning point for remediation with adults seeking assistance 
with literacy providers. 
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To date, most of the work has been concentrated on objective #1 and has 
included interface prototyping, knowledge-base development, and the creation of 
literacy scenarios, i.e., real-life activities in which literacy plays a central role. 
Work on objectives #2 & 3 are in the initial stages. 

The fact that technology for continuous speech recognition is not yet 
available has necessitated that the project move ahead more quickly on other 
fronts and develop plans for increasing the scope of the diagnostic capabilities. 
This increased scope and its attendant project plan is described fully in the 
following section. The separate development of the speech analysis component 
for the ALE is described in section IX. 

A final caveat to the project objectives is the extent to which the 
technologies of virtual reality can be fully integrated into the project. Although 
NASA’s Software Technology Branch is spearheading research in this area, the 
technology is still some years away from being easily incorporated into an 
application such as the ALE. Therefore, in order to create a virtual environment 
of authentic literacy materials and behavior, the project team is currently 
experimenting with a variety of technologies such as live and digitized video 
pictures with the capability of the interface to depict natural movement in each 
scene. 

HI. ALE: A Model for Authentic Assessment - Project Development 

Authentic, contextualized assessment is a complex issue. Yet several 
conceptual hurdles have been successfully maneuvered so far by the Project 
Team. This section will discuss critical break-throughs made in four major 
components of the ALE: (a) Scenario development, (b) literacy element 
knowledge base, (c) diagnostic continuums, and (d) comprehensive profile 
statements. 

A. Scenario Development 

L Multi-leveled assessment. The role of the scenarios was to offer a 
situation in which a client could enact real literacy actions embedded in 
purposeful print environments. As a diagnostic tool, the ALE must be able to 
assess a wide variety of literacy competencies. Unlike other assessment tools 
which measure a very restricted range of ability, the ALE assumes that adult 
readers' skills and strategies vary a great deal. Thus, the ALE must be a flexible, 
responsive system. Also, it is important that the environmental task demands 
quickly adjust to the clients without their awareness due to the stigma attached to 
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poor literacy performance. The project team tackled this problem by creating 
the notion of multi-layered scenarios rather than each having a specific 
skill/strategy level. This means two clients working next to each other could both 
be in the grocery store scenario, yet one could be interacting with print at a high 
level of sophistication such as filling out an application for a grocery card and 
another could be working at a much lower level, for example, recognizing ^ 
familiar logos on food items. The system will be created so that the clients 
response to each action will determine the rest of the scenario. While preventing 
cert ain scenarios from being labeled as "low," this also means that frustration 
levels can be kept to a minimum (which is a critical issue with adults who have 
failed academically) and as clients gain new strategies and skills, they will 
encounter new print challenges within a previously experienced scenario. 

2. After-scenarios . Another obstacle emerged within the scenario 
component. While much can be inferred from a client’s interactions with print, 
there are large arenas of literacy knowledge, critical to traditional analysis, which 
could not be tapped in the scenario as originally envisioned since speech analysis 
is not yet available. The project team wrestled with ways to assess more realms 
of literacy knowledge without speech analysis and recognition. As a solution to 
this dilemma, the idea of after-scenarios was developed. Following each series 
of context ualiz ed actions, such as renting a movie in a video store, In a separate 
simulation, the client will be asked more questions about the print experienced in 
that environment. In this clinical-type interview, the ALE might prompt a client 
to summarize the blurb on the back of a video or compare three signs in the store 
and determine which is the most informative. If clients want to see the document 
back in the original context, they just click on a certain place on the screen and 
the document reappears as it did in the scenario. The scenario and after-scenario 
partnership allows for the authenticity of real-life literacy events and the indepth 
probe of a clinical interview. Currently there is no other assessment tool of 
which the project team is aware in the education field which combines these two 
features. 

B. Development of the knowledge-base 

1. Grounding theory through field study . The field of literacy has yet to 
agree upon a definition of literacy let alone upon the range of literacy elements 
involved in making literate decisions. So Dr. Yaden thoroughly reviewed the 
literature in the field extracting individual literacy elements as they were 
described. These were then organized logically under traditional key features 
such as comprehension and fluency, to offer a comprehensive pool of literacy 
components no where else previously compiled. The next step was to reconnect 
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the elements pragmatically, in other words, reorganizing the elements so as to 
reflect a model of literacy which not only includes the components of reading 
and writing but also the grounded relationships between the components. After 
reviewing case studies and other work in the field, Drs. Yaden and Brown 
determined that the information they were seeking was not in the literature and 
thus it is deemed necessary to undertake a study to further document adult 
illiteracy. The current plan is to create a grounded theory, that is, a theory of 
adult literacy that is actually based on observations and interactions with disabled 
readers in the environment. This grounding will ensure the quality and 
marketability of the ALE as a revolutionary diagnostic tool. 

2. Client informants . The project team is also proposing an innovative 
design for software development by creating a team of client informants to work 
as fellow researchers to create the literacy model, scenarios, after-scenarios, and 
the interface. Rather than building a system and then testing it, the team has 
decided to work with the informants during the development process. This is 
critical with a population such as disabled readers who for various circumstances 
often lead very different lives than the project team members. Scenarios and 
after-scenarios will be created with the disabled readers' input from start to 
finish, thus avoiding development of inappropriate tasks. The transition to field 
testing will be easier due to participation and advice from disabled readers. 
Again, this is another consideration undertaken by the project team that will 
impress a business partner. 

C. Diagnostic Continuums 

1. Expanded scope of literacy assessment . Traditional survey and 
diagnostic tests in literacy generally measure students' skills in three areas: 
vocabulary, comprehension, and word recognition. Other facets of reading and 
writing processes such as metacognition, use of strategies, and literacy concepts 
are rarely even acknowledged. Since adult illiterates’ strengths are often in these 
unassessed domains, the project team faced another hurdle. The few 
unconventional tests that do attempt to get at these issues target very young 
children or are based on oral reading (as mentioned before a highly sophisticated 
speech analysis system would be necessary for this). The project team 
approached this problem by utilizing Dr. Brown’s previous work which identified 
expanded typologies of literacy performance and continuums of behavior as 
opposed to discrete skill levels only. In this earlier study, she documented 
different dimensions of children's literacy behavior based on observational data 
and product analysis. Thus after considering this work, Drs. Yaden and Brown 
decided to create a multi-dimensional profile represented by a series of 
continuums that will reflect not only traditional skills such as vocabulary and 
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comprehension, but also reader strategies and concepts of reading and writing. 
The ALE will break new ground in adult literacy diagnoses with a profile that 
both details outcome (i.e., performance level) and actions (i.e., strategies and 
their accompanying knowledge base). Another unusual facet of the ALE 
continuums is that they will be created through field work with disabled readers. 
By offering a multi-dimensional profile that reflects critical facets of adult 
literacy, the ALE promotes a model of remediation which is based on growth and 
assumes the importance of the socio-psycholinguistic aspects of literacy. 

D. Comprehensive Profile Statements 

1. Summary of diagnostic interpretations and continuums . In addition to 
the diagnostic statements interpreting the client's behavior on each of the 
continuums of literacy performance, the project team has developed the notion of 
comprehensive profile statements or summaries which will be available to give an 
overview of the adult's performance across all dimensions of literate behavior. 
Just as the diagnostic continuums summarize across individual literacy elements, 
so do the comprehensive statements give a perspective within which all aspects of 
the adult readers' performance on the ALE can be evaluated. The comprehensive 
summaries will integrate the adult client's behavior across, for example, 
performance on textual material, strategies for approaching literate tasks, and 
concepts about reading and writing events. Therefore, the exemplar lessons 
supplied to volunteer tutors will be based upon a much broader assessment of the 
adult's strengths and weaknesses than any current assessment available. 

IV. Conceptual Model of the Adult Literacy Evaluator 

Figure 1 depicts the current theoretical model of the ALE. The client will 
go through a series of actions in various literacy scenarios. 

Each scenario will involve many natural literacy interactions based on authentic 
and common life activities, such as going shopping or renting a video. 

Monitoring each action of the client, the system will adjust the following literacy 
demands appropriately. Then an after-scenario will utilize the print materials 
from that environment to allow for a clinical interview. By presenting specific 
questions about reading and writing, this component will reveal domains of 
literacy that would otherwise not be assessed during authentic actions. 

Each behavior will be analyzed by an referencing algorithm, possibly 
fuzzy logic. Actions will be related to identified literacy elements. Another 
referencing system will distribute the elements to diagnostic continuums. These 
continuums will be based on several broad literacy domains such as print 
interaction and print-coping strategies. A final referencing algorithm will 
analyze each continuum and connect these to one comprehensive profile statement 
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which will, in turn, be tied to a set of exemplar lessons. 

V. Project Plan and ALE Roadmap 

The current project plan is represented in Figure 2 and described in detail 
in the following narrative. 

The ALE road map is split into four regions: (a) Field Research refers to 
work that the literacy team will do in the field to gather in-depth data about the 
processes and conceptions of adult illiterates; (b) Theory and Literacy Model 
Development describes the theoretical and conceptual milestones; (c) Software 
Development details the programming demands; and (d) Grants and Quest for 
Business Partner relays the on-going search for external funding and a business 
that will want to market and sell the ALE at the completion of this project. 
Academic seasons are marked along the road map. Each season will be described 
below. 

Summer 1992 

Several fronts will move along this summer. The appropriate avenues at 
the university will be taken to gain consent for the field work to insure readiness 
for the fall. Research will continue on the diagnostic co ntinuums and an after- 
scenario will be developed. The project team will work together to explore new 
interface possibilities and create a field prototype by summer's end. Writing a R 
& D grant for the National Literacy Institute will be a time-demanding task 
which will occupy much of July and early August. This proposal is due August 
17, 1992. Efforts will continue to work with Joe Agosta, Director of 
Development at UH to find external funding. 

Fall 92/ Spring 93 

During this phase of the project, Drs. Yaden and Brown will conduct a 
research study on adult disabled readers. During the month of September fifteen 
tutoring sessions will be observed and the tutor and clients in each partnership 
will be interviewed. From this broad group, six pairs will be chosen for 
intensive study over the next eight months. Through observing tutoring sessions 
and spending time with the disabled readers as they go about their day-to-day 
activities, a grounded knowledge base will develop. Drs. Yaden and Brown will 
also begin tutoring their own clients as to have an inside perspective on the 
process of remediation. During the field work, pre liminar y development on the 
scenarios, after-scenarios, exemplar lessons, and referencing algorithm will 
continue. 
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Data analysis, occurring throughout the study, will reveal critical issues in 
remediation which have not been documented previously. This field work will 
insure the marketability of the ALE by allowing for the literacy team to create a 
product grounded in the needs and sensitive to the environmental constraints of 
adult remediation. Interface testing will enable the refinement of the prototype 
so that a Field Prototype 2 can be created as the field work is ensuing. It is 
critical to note that software development will be guided and continually refined 
by the data from the field. The search for external funding will continue as will 
the exploration of possible business partners. 

Summer 93 

From the field research, a few disabled readers will be selected to form an 
inf ormant group who will work intensely with the research team on several 
fronts: (a) completing the data analysis; (b) specifying interrelationships of 
literacy elements; (c) assessing and refining the Field Prototype 2; (d) finalizing 
the simulation; and (e) finalizing the continuums for the diagnostic profile. With 
disabled readers as essentially co-researchers immediate feedback will be 
available and misfires prevented. Components of the ALE will be finalized with 
the clients present; while being time effective, this should also be a selling point 
with a business partner. 

Fall 93 

During this phase the programmers will utilize the information gathered 
during the previous year to implement the simulation. The literacy team will 
design detailed scenarios, finalize the simulation to the continuum algorithm and 
finalize the diagnostic profiles. Hopefully, at this point a business partner will 
have been secured, if not, the search will intensify. 


Spring/ Summer 94 

While the literacy team is finalizing the inferencing algorithm from the 
continuums to the diagnostic profiles, writing exemplar lessons (potentially 
sixty), and completing each of the after-scenarios, the programmers will be 
implementing the rest of the ALE components. 
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Fall 94 


This final phase will involve testing in the field, further refinement, and 
writing of the documentation. The business partner will be integral to this phase 
as hopefully they would supply monies for field testing. This phase will conclude 
with a software program ready to be shrink-wrapped and handed off to a business 
to market and produce. 


VI. Presentations 

A. Justice Department, JSC. 

On March 12, 1991, a presentation of the project was given to a group of 
Justice Department officials by James Villarreal, Bob Way and David Yaden at the 
Johnson Space Center. In this presentation, the project's focus as an assessment 
and diagnostic tool incorporating the concepts of virtual reality was highlighted 
in comparison to an earlier conception of the system as being a tool for assisting 
the correct pronunciation of words. At that meeting, the first mock-up 
demonstration of the interface was presented which included live video, closed- 
caption TV and digitized pictures. A demonstration of the current speech 
recognition system being developed by Speech Systems Incorporated was given as 
well. The redefinition of the project as an assessment tool incorporating NASA's 
ICAT technology and speech recognition was well-received by those in 
attendance, and several justice department members volunteered federal prison 
sites for field-testing. 

B. Adult Literacy and Technology Conference, Costa Mesa, CA. 

On July 17-20, 1991, David Yaden and Bob Way made a presentation at the 
Adult Literacy and Technology Conference in Costa Mesa, California. This 
presentation included the rationale for and description of the simulation-based 
conception of the project as well as technical information about the research and 
development platform and the projected final computing requirements. From 
observation of other CAI programs on display at die conference, it was observed 
that the ALE is unique in its attempt to assess literacy in context and for the 
provision of intelligent feedback to both the client and literacy tutor. These 
features prompted several inquiries about the project both during and after the 
conference. 
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C. Representatives from state correctional institutions, JSC. 

On November, 1991 at JSC, the project was presented in detail to 
representatives of the National Institute of Corrections, the NASA Headquarters 
Technology Utilization Office, the Iowa Department of Corrections, Arizona 
State Department of Education and Garcz, McGing and Associates. At this 
meeting both Iowa and Arizona expressed a high degree of interest in the project 
because of its plan to incorporate real-life literacy skills as the focus of the 
assessment and for its provision of diagnostic interpretations and suggestions for 
remediation. Both states agreed to provide prison field-testing sites. 

At this meeting as well, the scope of the ALE was outlined and understood 
to be at least three years away from completion or at a point where it could be 
handed off to a commercial company for marketing. Primary in this projected 
timeline was the fact that speech recognition technology is unable at present to 
analyze continuous speech, a feature necessary for the oral reading analysis 
component of the ALE. The meeting participants agreed to collaborate on up- 
coming grants from the Department of Education (DOE) depending upon the 
status of the ALE at the time funding was available. 

VII. Publications 

Listed below in order of most recent publication date are articles 
disseminated by the project team describing ALE. These sources include a 
newspaper, conference proceedings, peer-reviewed journal and professional 
organization newsletter. 

Yaden, D. B., Brown, L. M., Way, R., & Villarreal, J. A. (in press). Authentic 
Literacy Assessment: NASA Technology Addressing Adult Illiteracy. 
Computers in the Schools. 

Yaden, D. B., & Brown, L. M. (1992). The Adult Literacy Evaluator 

NASA's Response to the Literacy Crisis. Mosaic: Research Notes on 
Literacy, 2 (2), 1 ,4. 

Villarreal, J. A., & Yaden, D. B. (1992). A NASA Spin-off: The Adult 
Literacy Tutor. Proceedings of the Adult Literacy and Technology 

Conference. 

Parker, F. (1992, March). Education Professor Using Today’s Technology to 
Help Fight Adult Illiteracy. UHOUSTON, March 6, p. 4. 
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Vni. External Funding 

To date, 30 federal and private funding agencies have been contacted by 
phone or letter describing the ALE and inquiring of their interest in financial 
support of the project. From these initial inquiries, 14 proposals have been sent 
to agencies interested and five (Kenan, Coca Cola, Coors, Casey & Beatrice) are 
still in various stages of review, according to spokespersons from those agencies. 
The project team is also working closely with the University of Houston 
Development Office and the Research Triangle Institute in following up on 
current proposals as well as in exploring new sources of f undin g not yet 
contacted. 

Department of Education funding 92-93 for corrections, originally 
anticipated, has not been applied for since the ALE will not ready as a 
demonstration project by September 30, 1992 when the funds from these grants 
will be disbursed. However, federal research and development monies for adult 
education and technology from the National Literacy Institute are available now 
for competitive applications (due August 14) and a major effort will be expended 
by the project team in preparing a proposal. 

IX. Speech Recognition Research 

In the ALE's original conception, speech analysis and recognition was to be 
one of the primary features of the system. To that end, two efforts have been 
underway to create an acceptable speech analysis component in order for the 
ALE to conduct an oral reading analysis, a fundamental aspect of any reading 
diagnosis. 

A. Speech Systems Incorporated. 

In the Spring of 1991, SSI delivered to JSC a pre liminar y version of the 
demonstration software with peripheral hardware to be run on a Sun platform. 
This system using a combination of scoring normalization techniques such as 
syntactic error modeling, score normalization and phoneme error modeling, 
resulted in a system which had an overall 80 % accuracy rate for target words, an 
80% accuracy rate for rejecting non-targets, but less than a 5% percent accuracy 
rate for recognizing all of the phonemes in a given word. Higher accuracy rates 
for this system are based on the recognition of one-syllable words, for 
recognizing all of the phonemes in a multi-syllable word, the system's accuracy is 
less that 3%. 
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Several aspects of the above system make it difficult to incorporate into the 
current project: (a) The cost of the Sun platform and necessary peripherals, (b) 
the fact that ALE is being designed on the Macintosh, and (c) the extremely low 
accuracy rates for determining all of the phonemes in both one-syllable and 
multi-syllable words. 

B. DKR Consulting. 

Currently, the goal of DKR is to develop a speech recognition and analysis 
system which will operate on a Macintosh II series computer with the first 
version of the system recognizing the 10 English vowel sounds in single syllable 
words. In this system, speech recognition is accomplished by using self- 
organizing feature maps (SOFM), learning vector quantization (LVQ), and 
hidden Maikov models (HMM) operating on digitized, preprocessed speech input. 
The only peripheral required in this system is a MacRecorder. The first version 
of the system is to be delivered in October 1992. 

DKR's system is best suited to the needs of ALE, partly because the 
developer worked on an early prototype of the system in the summer of 199 1 and 
knows the speech analysis requirements of the system. However, the three most 
immediate difficulties are (a) the first version will not be delivered until October 
1992, thus requiring the first prototype of ALE to be without speech recognition 
capabilities, (b) it will recognize vowels only and, (c) the developer cannot work 
full-time on the project. 
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Adult Literacy Evaluator Conceptual Model 
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